Systems, apparatus, and methods for autonomous tripping of well pipes

ABSTRACT

A robotic system coupled to a racking platform of an oil well service or drilling rig comprising a base coupled to the racking platform at a fixed location, a mast pivotally coupled to the base by a mast pivot joint allowing rotation of the mast about a mast axis, a mast actuator for controllably rotating the mast about the mast pivot joint, an arm coupled to the mast and moveable along a radial direction with respect to the mast axis, an arm actuator for controllably moving the arm along the radial direction, an end effector pivotally coupled to an end of the arm by an end effector pivot joint allowing rotation of the end effector about an end effector axis oriented generally parallel to the mast axis, and an end effector actuator for controllably rotating the end effector about the end effector pivot joint. The end effector comprises at least one grabbing member operable to selectively grab an elongated object under control of a grabbing member actuator.

REFERENCE TO RELATED APPLICATION

This is a continuation application of co-pending PCT/CA2007/001054,filed Jun. 14, 2007, which claims Paris Convention priority from U.S.Patent Application No. 60/804,753, filed on 14 Jun. 2006, the contentsof each prior application being hereby incorporated herein in itsentirety by express reference thereto.

TECHNICAL FIELD

This invention relates to manipulation of elongated objects, and certainembodiments relate to servicing oil wells. Particular embodiments of theinvention provide systems and methods for autonomous tripping of oilwell pipes.

BACKGROUND

One of the most hazardous tasks in industry is servicing oil wells toperform maintenance and/or repair operations on the oil wells. Oil wellservicing involves removal of oil pipes from the ground (tripping out)and subsequent re-insertion of oil pipe into the ground (tripping in).Presently, oil well servicing requires significant human involvement andexposes workers to serious health and safety risks. Typical oil rigservicing systems require: a rig operator, who operates the elevatorwhich lifts the pipe out of the ground and lowers the pipe into theground; a ground operator, who handles the pipes that are being hoistedby the elevator and places the lower ends of the pipes into a drip tray;and a derrick man, who works on a raised platform (typically 20-55 feetabove the ground) to manipulate the upper ends of the pipes into anupper racking board.

Oil well servicing involves a number of dangers, particularly for thederrick man on the raised platform. The raised platform on which thederrick man works is sometimes referred to colloquially as a “monkeyboard” because of its location well above the ground and the dangersposed to operators working thereon. Accidents during oil well servicingoperations are costly to equipment and human lives and can damage thepublic image of the oil industry.

Protecting human lives in hazardous industrial applications has longbeen a foremost concern of industry. The inventors have determined thatthere exists a need to automate some of the tasks involved in oil wellservicing and to provide systems for autonomously performing some ofthese tasks.

SUMMARY

The following embodiments and aspects thereof are described andillustrated in conjunction with systems, tools and methods which aremeant to be exemplary and illustrative, not limiting in scope. Invarious embodiments, one or more of the above-described problems havebeen reduced or eliminated, while other embodiments are directed toother improvements.

One aspect of the invention provides a robotic system coupled to aracking platform of an oil well service or drilling rig. The roboticsystem comprises a base coupled to the racking platform at a fixedlocation, a mast pivotally coupled to the base by a mast pivot jointallowing rotation of the mast about a mast axis, a mast actuator forcontrollably rotating the mast about the mast pivot joint, an armcoupled to the mast and moveable along a radial direction with respectto the mast axis, an arm actuator for controllably moving the arm alongthe radial direction, an end effector pivotally coupled to an end of thearm by an end effector pivot joint allowing rotation of the end effectorabout an end effector axis oriented generally parallel to the mast axis,and an end effector actuator for controllably rotating the end effectorabout the end effector pivot joint. The end effector comprises at leastone grabbing member operable to selectively grab a elongated objectunder control of a grabbing member actuator.

Another aspect of the invention provides a mobile apparatus for oil wellservicing. The apparatus comprises a mobile platform, a derrickpivotally coupled to the mobile platform and moveable between a deployedposition and a storage position, a racking platform defining a pluralityof elongated object receiving locations coupled to the derrick, anelevator supported from the derrick for raising and lowering elongatedmembers along an elevator axis, and, a robotic system coupled to theracking platform at a fixed location, the robotic system comprising amechanism having at least three degrees of freedom for manipulating anupper portion of an elongated member within a plane generally parallelto a plane of the racking platform.

Further aspects of the invention and features of specific embodiments ofthe invention are described below.

BRIEF DESCRIPTION OF DRAWINGS

In drawings which show non-limiting embodiments of the invention:

FIG. 1 is a schematic side plan view of an automated oil well trippingsystem according to a particular embodiment of the invention;

FIGS. 2A, 2B and 2C respectively represent side, top and side views ofthe robotic system of the FIG. 1 tripping system in variousconfigurations;

FIG. 2D is an isometric view of an end effector according to aparticular embodiment of the invention;

FIGS. 2E-G show internal links of the end effector of FIG. 2D in variouspositions;

FIGS. 3A and 3B respectively represent side and top plan views of therack and the robotic system of the FIG. 1 tripping system;

FIGS. 4A and 4B respectively represent top and side views of the rack ofthe FIG. 1 tripping system;

FIGS. 5A, 5B and 5C respectively represent partial top, side andcross-sectional views of the rack of the FIG. 1 tripping system;

FIG. 5D is an exploded view of a finger member of the rack of the FIG. 1tripping system;

FIGS. 5E-5I represent top plan views of a pipe being inserted into therack of the FIG. 1 tripping system;

FIG. 5J represents a top plan view of a portion of the rack of the FIG.1 tripping system after it has been filled with pipes;

FIGS. 6A, 6B and 6C schematically depict the steps involved in atripping out operation according to a particular embodiment of theinvention;

FIGS. 7A, 7B and 7C schematically depict the steps involved in atripping in operation according to a particular embodiment of theinvention;

FIG. 8 schematically depicts an image sensing and robot control systemaccording to a particular embodiment of the invention;

FIG. 9 schematically depicts other elements of the FIG. 8 system;

FIGS. 10A-10C depict image preprocessing steps according to a particularembodiment of the invention;

FIGS. 11A, 11B and 11C respectively depict image data, verticalprojections of the image data and horizontal projections of the imagedata according to a particular embodiment of the invention;

FIG. 11D is a plot showing a curvelet which may be convolved with theFIG. 11C horizontal projections to determine the vertical position ofthe top of the pipe;

FIG. 12 is a schematic depiction of a cross-correlation templatematching technique for locating the top of a pipe according to aparticular embodiment of the invention;

FIGS. 13A, 13B and 13C schematically depict a vertical projection,feature recognition technique for locating a second point on the pipeaxis and thereby determining the orientation of the pipe;

FIGS. 14A-14C schematically depict an edge detection process that may beused to generate binary edge detection information for inputting into aHough transform;

FIGS. 15A-15G schematically depict a technique for determining suddenchanges in acceleration which may be indicative of the bottom of thepipe impacting the drip tray;

FIG. 16A depicts a method for tripping out a pipe according to aparticular embodiment of the invention;

FIG. 16B depicts a method for tripping in a pipe according to aparticular embodiment of the invention;

FIG. 17 schematically depicts a robot control system according toanother embodiment of the invention

FIG. 18 depicts a method for tripping out a pipe according to anotherembodiment of the invention;

FIGS. 19A-D schematically depict steps involved in the tripping outoperation according to the embodiment of FIG. 18; and,

FIGS. 20A and 20B schematically depict a portion of an elevatoraccording to one embodiment of the invention.

DESCRIPTION

Throughout the following description specific details are set forth inorder to provide a more thorough understanding to persons skilled in theart. However, well known elements may not have been shown or describedin detail to avoid unnecessarily obscuring the disclosure. Accordingly,the description and drawings are to be regarded in an illustrative,rather than a restrictive, sense.

FIGS. 1-5C schematically depict a system 10 for autonomously performingportions of the tripping (in and out) operations involved in oil wellservicing in accordance with a particular embodiment of the invention.In the illustrated embodiment, system 10 is a mobile system which iscapable of servicing different oil wells. To achieve this mobility,system 10 has a relatively lightweight construction in comparison toexisting oil well servicing systems, and is supported by a mobileplatform E1. Mobile platform E1 may be towed by a truck, tractor orother suitable vehicle. It is not generally necessary that system 10 ismobile. System 10 may be associated with and used to service aparticular oil well.

Mobile platform E1 supports a derrick E2. Preferably, derrick E2 ispivotally coupled to platform E1, such that derrick E2 may be pivotedbetween a generally vertical orientation (shown in FIG. 1) and agenerally horizontal orientation (not shown) atop mobile platform E1.Derrick E2 supports an operating platform E4 and a racking platform N1.Derrick E2 may comprise a derrick extension E3 to which racking platformN1 is coupled. In some embodiments, racking platform N1 may be pivotallycoupled to derrick E2 such that racking platform N1 may be pivoted to begenerally parallel to derrick E2 when derrick E2 is in the generallyhorizontal orientation to facilitate transportation of system 10.

In typical embodiments, when derrick E2 is in its generally verticalorientation, operating platform E4 is located less than 10 feet abovethe ground (or above the top of an oil well) and racking platform N1 maybe located between 20 and 80 feet above operating platform E4. In someembodiments, the position of derrick extension E3 is adjustable alongthe length of derrick E2, such that the location of racking platform N1is adjustable. The location of operating platform E4 may also beadjustable.

Derrick E2 also supports a crane system E6, which may be referred to asan “elevator”. Elevator E6 comprises a pipe coupler E8 for coupling tooil well pipes 30. Elevator E6 also comprises a suitable actuator (notshown) for moving pipe coupler E8 (and any pipe 130 to which it iscoupled) upwardly and downwardly along the general direction of elevatoraxis E11. Elevators are well known in the field of oil well servicingand are not explained further herein.

System 10 comprises a robotic system N2 which is mounted to rackingplatform N1. Robotic system N2 may be mounted at a fixed location onracking platform N1. As discussed in more detail below, robotic systemN2 is configured to interact with an upper portion of an elongatedobject such as, for example, an oil well pipe 130, such that a humanbeing is not required on racking platform N1 to perform trippingoperations. In some embodiments, robotic system N2 comprises a mechanismhaving at least three degrees of freedom for manipulating an end of anelongated object within a plane generally parallel to a plane of rackingplatform N1. System 10 also comprises one or more suitably programmedsystem controllers (not shown in FIGS. 1-5C) for controlling theoperation of robotic system N2.

FIGS. 2A-2C schematically depict more detail of a robotic system N2according to a particular embodiment of the invention. In general,robotic system N2 comprises a mechanism for controllably moving an endeffector N7 capable of engaging or otherwise interacting with pipe 130.In some embodiments, robotic system N2 makes use of one or more sensorsto determine one or more positional characteristics of pipe 130. Suchsensors may comprise, for example, laser sensors, ultrasonic sensors ormagnetic sensors. In some embodiments, robotic system N2 may bepreprogrammed with known positional characteristics of pipe 130.

Robotic system N2 also makes use of one or more sensors to determine oneor more positional characteristics of end effector N7. Based on thepositional characteristics of pipe 130 and end effector N7, roboticsystem N2 may cause end effector N7 to autonomously engage and disengagepipe 130 to perform tripping operations. When pipe 130 is engaged by endeffector N7, robotic system N2 may controllably manipulate the positionof end effector N7 and thereby controllably manipulate the position ofpipe 130.

In the illustrated embodiment, robotic system N2 comprises a manipulablerobot arm N6 coupled to an elongated mast 104. End effector N7 iscoupled to an end of arm N6 opposite mast 104. As shown in FIGS. 2A-2C,arm N6 may comprise a mechanical assembly having a plurality of segmentsmoveably coupled to one another to facilitate movement of end effectorN7 in along a radial direction shown by double-headed arrow 102. Thisradial movement of arm N6 provides robotic system N2 with a first degreeof freedom.

In the illustrated embodiment, arm N6 comprises segments 106, 106A and109. Segments 106 and 109 are each pivotally coupled to mast 104 atinner (i.e., closer to mast 104) ends thereof. Segment 109 is pivotallycoupled to a middle portion of segment 106, and segment 106A ispivotally coupled to the outer (i.e., farther from mast 104) end ofsegment 109. Segments 106 and 106A are coupled to a pivot joint 112 atthe end of arm N6 to which end effector N7 is coupled, such that therelative orientation between mast 104 and end effector N7 is maintainedas arm N6 moves along the radial direction. FIG. 2A shows how therelative orientation between mast 104 and end effector N7 is maintainedwhen arm N6 is retracted toward mast 104 and extended away from mast104. As shown in FIG. 2A, when mast 104 is generally verticallyoriented, end effector N7 is generally horizontally oriented.

In the illustrated embodiment, mast 104 houses a suitable arm actuator105. In some embodiments, the arm actuator 105 may comprise, forexample, a servo motor, another type of motorized actuator, or ahydraulic actuator. The arm actuator 105 is capable of moving armsegment 106 of arm N6 along the elongated dimension of mast 104. Whenthe arm actuator 105 moves arm segment 106 toward arm segment 109 (e.g.downwardly in FIG. 2A), arm N6 causes end effector N7 to extend awayfrom mast 104. Conversely, when the actuator 105 moves arm segment 106away from arm segment 109 (e.g. upwardly in FIG. 2A), arm N6 causes endeffector N7 to be withdrawn toward mast 104. Other mechanisms andactuators could be used to implement arm N6 and to provide thefunctionality described herein.

Robotic system N2 also comprises one or more sensors (not specificallyenumerated) capable of detecting information which enables the systemcontroller to determine the current configuration/position of arm N6(and/or the position of end effector N7) relative to mast 104. Suchsensors may comprise one or more encoders coupled to one or more of thejoints of arm N6, one or more sensors coupled to the arm actuator whichcauses arm N6 to move and/or one or more other suitably configuredsensors. Those skilled in the art will appreciate that the systemcontroller may be programmed with a model of arm N6, such that theinformation provided by such sensors may be used to determine thecurrent configuration/position of arm N6 (and/or end effector N7).

End effector N7 is pivotally coupled to the end of arm N6 by an endeffector pivot joint 110 to allow pivotal movement of end effector N7 inthe directions shown by double-headed arrow 108 (FIG. 2B). This pivotalcoupling of end effector N7 to arm N6 provides robotic system N2 with asecond degree of freedom. Robotic system N2 comprises an end effectoractuator (see FIG. 2D) for manipulating end effector N7 about pivotjoint 110. The end effector actuator may comprise, for example, a servomotor or some other type of actuator.

End effector N7 comprises at least one grabbing member operable toselectively grip an elongated object such as, for example, pipe 130. Inthe illustrated embodiment, end effector N7 comprises a pair ofopposable grabbing members 107A, 107B which are shaped for grasping anoil well pipe 130 around a portion of its circumferential surface.Grabbing members 107A and 107B may be selectively opened and closed by agrabbing member actuator located within end effector, under control ofthe system controller. The inner surfaces of grabbing members 107A and107B may be curved and/or angled to fit around the circumferentialsurface of oil well pipe 130. In other embodiments, end effector N7 maytake other forms that provide the functionality described herein.

FIGS. 2D-G show more details of end effector N7 according to aparticular embodiment. Various components of end effector N7 are omittedor depicted transparently in FIGS. 2D-G so that internal componentsthereof may be shown. As shown in FIG. 2D, an end effector actuator 111is coupled between pivot joint 112 and pivot joint 110 for manipulatingend effector N7 about pivot joint 110. End effector actuator 111 maycomprise, for example, a harmonic drive coupled to a reducing gearbox.End effector actuator 111 is typically covered by a cylindrical cover(not shown in FIG. 2D). A mechanical switch 113 may be positionedbetween grabbing members 107A and 107B, which is activated when anelongated object is received between grabbing members 107A and 107B toprovide the system controller with an indication that the elongatedobject is in position for grabbing. Instead of or in addition tomechanical switch 113, ultrasonic, infrared, magnetic or other sensorsmay be provided for detecting the presence of a pipe 130 betweengrabbing members 107A and 107B.

As shown in FIGS. 2E-G, grabbing members 107A and 107B are pivotallycoupled to a housing of end effector N7 by fixed pivot joints 107C and107D. Fixed pivot joints 107C and 107D may comprise rubber bushings 107Hor the like to absorb shocks generated from a pipe contacting grabbingmembers 107A and 107B. Grabbing members 107A and 107B are coupled to agrabbing member actuator 119 by means of pivoting links 107E and 107Fand an extendable member 107G. Grabbing member actuator 119 maycomprise, for example, a stepper motor, another type of motorizedactuator, or a hydraulic actuator.

In the illustrated embodiment, grabbing member actuator 119 may extendextendable member 107G to move grabbing members 107A and 107B into anopen position, as shown in FIG. 2E, and may retract extendable member107G to move grabbing members 107A and 107B into a closed position, asshown in FIG. 2G. When in the closed position, pivoting links 107E and107F are positioned to oppose any opening of grabbing members 107A and107B, such that end effector N7 is self-locking.

Grabbing members 107A and 107B may be detachable in some embodiments, sothat different fingers may be provided to allow end effector N7 to grippipes having different diameters. This permits grabbing member actuator119 to move through the same range of motion to move grabbing members107A and 107B between the closed and open positions for different pipes.In some embodiments, grabbing members 107A and 107B may be selected suchthat there is approximately ⅛th of an inch clearance between the innersurfaces of grabbing members 107A and 107B and a pipe when grabbingmembers 107A and 107B are in the closed position shown in FIG. 2G.

Robotic system N2 also comprises one or more sensors (not specificallyenumerated) capable of detecting information which enables the systemcontroller to determine the current configuration/position of endeffector N7 relative to arm N6 and/or mast 104 and the current positionof grabbing members 107A and 107B relative to end effector N7 and/or toone another. Such sensors may comprise encoders coupled to one or moreof pivot joints 110, 112 and/or the pivot joints within end effector N7,sensors coupled to end effector actuator 111 and/or grabbing memberactuator 119, or other suitably configured sensors. In some embodiments,sensors may also be provided for detecting torque on end effector N7and/or grabbing members 107A and 107B. Those skilled in the art willappreciate that the system controller may be programmed with a model ofend effector N7, such that the information provided by such sensors maybe used to determine the current configuration/position of end effectorN7 and grabbing members 107A and 107B.

Returning to FIGS. 2A-C, robotic system N2 comprises a base 115 coupledto a fixed location on racking platform N1. Mast 104 is pivotallycoupled to base 115 by a pivot joint N8 to allow pivotal movement ofmast 104 (and arm N6) about a mast axis 117 in the directions shown bydouble-headed arrow 114 (FIG. 2B). This pivotal coupling providesrobotic system N2 with a third degree of freedom. Robotic system N2comprises a mast actuator (not specifically enumerated) for manipulatingmast 104 about pivot joint N8. The mast actuator may comprise, forexample, a servo motor, a harmonic drive and a reducing gearbox, anothertype of motorized actuator, or a hydraulic actuator. Robotic system N2also comprises one or more sensors for detecting the position of mast104 about pivot joint N8. These sensors may comprise one or moreencoders coupled to pivot joint N8, one or more sensors coupled to themast actuator or one or more other suitably configured sensors.

Base 115 of robotic system N2 may be pivotally coupled to rackingplatform N1 by a pivot joint 116 for pivotal movement of robotic systemN2 in the directions shown by double-headed arrow 118 (FIG. 2C). In theillustrated embodiment, a hydraulic actuator N4 is provided formanipulating robotic system N2 about pivot joint 116 between anoperating position (FIG. 2A), wherein mast 104 extends generallyperpendicularly to the plane of racking platform N1 and a storageposition (FIG. 2C), wherein mast 104 lies generally within the plane ofracking platform N1. In other embodiments, actuator N4 may comprise adifferent type of actuator (e.g. a motorized actuator). Robotic systemN2 may also comprise one or more sensors for detecting the position ofrobotic system N2 about pivot joint 116. These sensors may comprise oneor more encoders coupled to pivot joint 116, one or more sensors coupledto actuator N4 or one or more other suitably configured sensors.

FIGS. 3A, 3B, 4A and 4B schematically depict racking platform N1 in moredetail. Racking platform N1 comprises an adjustable pipe rack N5. RackN5 securely stores oil well pipes 130 after they are removed from an oilwell or before they are inserted into an oil well. In the illustratedembodiment, rack N5 comprises a number of slidably adjustable pipe rackfingers N9, N10 mounted on a frame of racking platform N1. On one side120 of racking platform N1, pipe rack fingers N9 are slidably adjustedsuch that their spacing (relative to one another) will accommodate pipeshaving a first diameter. On the opposing side 122 of racking platformN1, pipe rack fingers N10 are slidably adjusted such that their spacing(relative to one another) will accommodate pipes having a seconddiameter. As shown in FIG. 4B, racking platform N1 may travel through anarc (shown by double-headed arrow 124) about a pivotal coupling 126 toderrick extension E3. A suitable actuator (not specifically enumerated)may be provided to effect this movement of racking platform N1 aboutpivotal coupling 126.

FIGS. 5A-D schematically depict adjustable pipe rack fingers N10 indetail. It should be understood that pipe rack fingers N9 aresubstantially similar to pipe rack fingers N10. Pipe rack fingers N10comprise a plurality of finger members N13. In the illustratedembodiment, finger members N13 are slidably mounted to racking platformN1 by adjustable coupling mechanism N11 and suitable fasteners N12.Finger members N13 may generally be coupled to racking platform N1 usingany suitable mechanism. Preferably, this coupling mechanism may compriseactuators N17A to provide adjustable spacing N17 between finger membersN13. In the illustrated embodiment, each finger member N13 comprises aplurality of concave pipe-receiving portions 132 for receiving a portionof the circumferential surface of a pipe 130. Concave pipe-receivingportions 132 may be arcuate.

A plurality of toggle locks N14 and N16 may be pivotally coupled (atpivot joints 134) to each finger member N13. Toggle locks N14 and N16may be held in place by retaining bars N18. Each toggle lock N14 may bearranged in a complementary pair with a corresponding one of togglelocks N16. In the illustrated embodiment, toggle locks N14 extend fromtheir respective pivot joints 134 toward an open end 133 of pipe rackfingers N10 (i.e. in the direction of arrow 142). In the illustratedembodiment, each toggle lock N14 comprises a concave pipe-receivingportion 136 shaped to receive a portion of the circumferential surfaceof a pipe 130. Concave portions 136 may be arcuate.

In the illustrated embodiment, each toggle lock N14 also comprises firstand second beveled portions 138, 139. First beveled portion 138 isshaped such that force applied against first beveled portion 138 in thedirection of arrow 141 will cause the corresponding toggle lock N14 topivot about its pivot joint 134 out of the path between finger membersN13 (i.e. in a counterclockwise direction in the FIG. 5A illustration).Second beveled portion 139 is shaped such that force applied against thesecond beveled portion 139 in the direction of arrow 142 will also causethe corresponding toggle lock N14 to pivot about its pivot joint 134 outof the path between finger members N13 (i.e. in a counterclockwisedirection in the FIG. 5A illustration). Toggle locks N16 aresubstantially similar to toggle locks N14, except that toggle locks N16are oriented in the opposite direction (i.e. they extend away from pivotjoints 134 in the direction of arrow 141) and toggle locks N16 arespaced apart from toggle locks N14 in the axial direction of pipes 130(see FIGS. 5C and 5D).

As best seen in FIG. 5D, a spring N15 may be coupled betweencorresponding pairs of toggle locks N14 and N16 to bias each pair oftoggle locks N14 and N16 into a predetermined angular relationship withone another. Each pair of toggle locks N14 and N16 may compriseinterlocking features 135 which limit the range of angular movementtherebetween. Each pair of toggle locks N14 and N16 except the “last”pair closest to coupling mechanism N11 (i.e., the pair farthest fromopen end 133) may be free to rotate about the corresponding pivot joint134. The last pair of toggle locks N14 and N16 may be provided with abiasing mechanism 137 (which may comprise, for example, a tension coilspring) for biasing the last toggle lock N16 into a pipe retainingposition wherein toggle lock N16 extends into the path between fingermembers N13 (i.e., in a counterclockwise direction in the FIG. 5Dillustration). Posts 134A may be provided on finger member N13 to limitthe range of motion of each pair of toggle locks N14 and N16 about pivotjoints 134. The concave pipe-receiving portions 136 of adjacent togglelocks N14, N16 from different pairs (other than the first toggle lockN14 and the last toggle lock N16) may overlap one another, such thattoggle locks N14, N16 operate in tandem to retain pipes 130 (except atthe ends of finger members N13), as described below with reference toFIGS. 5E-J.

FIGS. 5E-5J illustrate how pipes 130 may be inserted into pipe rackfingers N10 according to a particular embodiment. As shown in FIG. 5E, apipe 130 is inserted into pipe rack fingers N10 between finger membersN13 from open end 133 (e.g. in the direction of arrow 141). As pipe 130is inserted it encounters the first beveled end 138 of a first togglelock N14. The pipe 130 being inserted causes the first pair of togglelocks N14 and N16 to pivot about pivot joint 134 to move toggle lock N14out of the path between finger members N13, as shown in FIG. 5F. Next,as shown in FIG. 5G, pipe 130 encounters second beveled end 139 oftoggle lock N16, which causes the first pair of toggle locks N14 and N16to pivot about pivot joint 134 to move toggle lock N16 out of the pathbetween finger members N13. This process continues until pipe 130reaches its racking location defined by one of the pipe receivingportions 132 on opposing finger member N13. If pipe 130 is the firstpipe being inserted between two adjacent finger members N13, pipe 130must be pushed with enough force to overcome biasing mechanism 137 to bemoved into its racking location, and the last toggle lock N16 retainsthe pipe in its racking location through the action of biasing mechanism137.

If pipe 130 is not the first pipe being inserted between two adjacentfinger members N13, the presence of a previously racked pipe 130 willrequire spring N15 to flex to allow toggle lock N14 to pivot out of theway, as shown in FIG. 5H. Once pipe 130 reaches its final rackingposition, toggle lock N14 will be forced back toward pipe 130 to retainpipe 130 in its final racking position, as shown in FIG. 5I, and thecorresponding toggle lock N16 will assist in retaining the previouslyracked pipe 130 in its racking position. Once pipe 130 reaches its finallocation, the bias forces provided by springs N15 cause pipe 130 to beretained between the concave portions 136 of the toggle locks N14, N16and a particular concave portion 132 on the opposing finger member N13.At the ends of finger members N13, a pipe 130 may be retained by asingle toggle lock N14 or by a single toggle lock N16. FIG. 5J shows aportion of pipe rack N5 filled with pipes 130. In some embodiments,toggle locks N14, N16 are provided with locking mechanisms (not shown)which allow them to lock once they receive pipes 130, such that togglelocks N14, N16 are prevented from pivoting when locked. Removal of pipes130 from pipe rack N5 requires overcoming the bias forces of springs N15and biasing mechanism 137 on toggle locks N14, N16, and may beaccomplished by sequentially pulling pipes 130 toward open end 133,starting with the pipe 130 closest to open end 133.

Referring to FIGS. 6A, 6B and 6C, the tripping out (removal) of oilpiping may proceed as follows in embodiments which comprise a visualserving system, as described further below. First, elevator E6 islowered to well head E5 and pipe coupler E8 is coupled onto a pipe 130at or near its upper end. Elevator mechanism E6 is then drawn upwardlyand with it pipe 130 (as shown in FIG. 6A), until the lower end of pipe130 is clear of well head E5. Next, a human drill head operator E10latches a rotary actuator (not shown) onto pipe 130 at or near its lowerend. The rotary actuator then unscrews pipe 130 from the pipe remainingin the well. Next, operator E10 disengages the rotary actuator from pipe130, leaving the lower end of pipe 130 free to move. Operator E10 thenguides the lower end of pipe 130 over a drip tray E9 and lowers elevatorE6, as shown in FIG. 6B. When the lower end of pipe 130 is positionedover the drip tray E9, the orientation of pipe 130 is no longervertical.

Next, robotic system N2 uses a visual serving system (not specificallyenumerated) to locate the upper end of pipe 130 and to autonomously andcontrollably position robotic system N2, arm N6 and/or end effector N7,such that end effector N7 is disposed to grip pipe 130 at or near itsupper end. End effector N7 then securely engages pipe 130, as shown inFIG. 6C. Once end effector N7 has securely engaged pipe 130, pipecoupler E8 is disengaged from pipe 130. Robotic system N2, arm N6 and/orend effector N7 are then moved so that the upper end of pipe 130 isplaced into pipe rack N5. The visual serving system, which allowsrobotic system N2 to locate the upper end of pipe 130 and to positionend effector N7 in a location where it can grip pipe 130, is explainedin more detail below.

Referring to FIGS. 1, 7A, 7B and 7C, the tripping in (insertion) of oilpiping may proceeds as follows. First, robotic system N2, arm N6 and/orend effector N7 are autonomously manipulated so that end effector N7 ispositioned to grip a pipe 130 held in pipe rack N5. Once end effector N7is positioned in this manner, end effector N7 securely engages pipe 130,as shown in FIG. 7A. Robotic system N2 then disengages pipe 130 frompipe rack N5. Robotic system N2, arm N6 and/or end effector N7 are thenautonomously moved so that the upper end of pipe 130 is brought intovertical alignment with the axis E11 of elevator E6. Next, elevator E6is lowered and pipe coupler E8 is coupled onto pipe 130 at or near itsupper end, as shown in FIG. 7B. Once pipe coupler E8 is securelyattached to pipe 130, end effector N7 is disengaged from pipe 130, asshown in FIG. 7C. Operator E10 then moves the bottom of pipe 130 fromdrip tray E9 into alignment with another pipe disposed inside the well.Next, operator E10 latches the rotary actuator onto the lower end ofpipe 130. The rotary actuator screws pipe 130 onto the pipe alreadyinside the well. Operator E10 then disengages the rotary actuator frompipe 130 and lowers elevator E6 and pipe 130 into the well to completethe tripping in operation.

As discussed briefly above, in some embodiments, oil well trippingsystem 10 makes use of a machine vision system for autonomouslycontrolling the movement of robotic system N2. The following paragraphsdescribe an example machine vision system according to a particularembodiment, but it is to be understood that different machine visionsystems could be used with system 10. In other embodiments, system 10may be used without a machine vision system, as described further below.

FIGS. 8 and 9 schematically depict a machine vision and robot controlsystem 200 according to a particular embodiment of the invention. Therack (not specifically enumerated) shown in FIG. 8 is different fromrack N5 shown in FIGS. 1-5C. The rack of FIG. 8 comprises concentricarc-shaped finger members (not specifically enumerated) which allow theinsertion of pipe 130 into the FIG. 8 rack by pivotal movement ofrobotic system N2 about pivot joint N8 (see FIG. 2B). In the illustratedembodiment system 200 comprises an image sensing system 202 and acontroller 210. Imaging sensing system 202 obtains image data 204 andprovides image data 204 to controller 210. Controller 210 interpretsimage data 204 to obtain a target position for end effector N7 duringtripping operations. Controller 210 uses image data 204 together withposition data 205 from the position sensors associated with roboticsystem N2 to generate suitable control signals 206 which control themovement of robotic system N2 so that end effector N7 achieves thedesired target position.

Image sensing system 202 obtains image data 204 relating to a region ina vicinity of elevator axis E11 above racking platform N1. Pipe 130 isexpected to pass through this region during tripping operations. In theillustrated embodiment, image sensing system 202 comprises a pluralityof image sensing devices 202A, 202B, 202C. Image sensing devices 202A,202B, 202C are spaced apart from one another and are oriented torespectively capture image data 204A, 204B, 204C in the region ofinterest. In one particular embodiment, image sensing devices 202A,202B, 202C may be digital cameras which make use of arrays of CCD orCMOS or similar optical detectors. In other embodiments, image sensingsystem may comprise a different numbers of image sensing devices.

In the illustrated embodiment, controller 210 comprises an imageprocessing component 212 which receives image data 204 from imagesensing system 202 and generates a target position d_(i) for endeffector N7. Determining the target position d_(i) of end effector N7may involve determining the position of the upper end of a pipe 130 inelevator E6 and the orientation of the pipe 130 relative to a known axis(e.g. elevator axis E11 or a horizontal axis). Controller 210 furthercomprises a robot unit inverse kinematic component 214, which processestarget position d_(i) to obtain a set of desired coordinates q_(d) forrobotic system N2 (in the measurement space of the position sensors ofrobotic system N2). Comparison component 215 then compares the desiredcoordinates q_(d) for robotic system N2 to the actual robot unitcoordinates q (i.e. robot unit position data 205 sensed by the sensorsof robotic system N2). Robot control component 216 then uses thedifferences between the actual coordinates q and the desired coordinatesq_(d) to generate appropriate control signals 206 for the actuators ofrobotic system N2.

Image processing component 212 may perform a number of imagemanipulation operations prior to (or as a part of) the process ofdetermining the target position d_(i) of end effector N7. In oneparticular embodiment, the processing operations performed by imageprocessing component 212 on incoming image data 204 comprise: optionallyprocessing color image data 204 (if necessary) to obtain intensityvalues of the pixels in the image; determining the mean pixel intensityvalue of the resultant image; subtracting the mean pixel intensity valuefrom the intensity values the pixels in the image; adding a pixelintensity offset value to the intensity value of the pixels in theimage; and applying a low pass filter to the image.

FIGS. 10A-10C depict an example of such image processing. Image data 300represents the intensity values of image data 204 obtained from imagesensing system 202. In some embodiments, image sensing system 202 maydirectly provide intensity value image data 300. Image data 300 includesa fair amount of background scenery which may make it difficult todetermine the location of the end 131 of pipe 130. Image processingcomponent 212 may process image data 300 to obtain image data 302 by:determining a mean intensity value of image data 300; subtracting themean intensity value from image data 300; and adding an offset thresholdvalue to reduce the darkness of the resultant image data. Image data 302is then further processed to obtain image 304 by applying a low passfilter to “smooth out” the image. In one particular embodiment, the lowpass filter is a Gaussian filter. It can be seen that background sceneryis largely eliminated from image data 304.

In some embodiments, image processing component 212 makes use of afeature detection process which operates on a projection of the imagedata to determine the position of the end 131 of pipe 130. Preferably,this feature detection process operates on one or more projections ofbackground-reduced image data 304. The projections on which imageprocessing component 212 performs the feature detection process may behorizontal, vertical or arbitrary projections. These projections may bedetermined on the basis of the field of view of the image, which may inturn depend on the position and orientation of the images sensors 202A,20B, 20C and an approximate expected position of pipe 130. To reduceprocessing time, image processing component 212 may identify a region ofinterest from within image data 304 based on an approximate expectedposition of pipe 130 and perform the feature detection process only ondata from the region of interest.

FIGS. 11A-11D schematically depict a feature detection process fordetermining the position of the end 131 of a pipe 130 according to aparticular embodiment of the invention. FIG. 11A depicts image data 304which has been processed to remove the background scenery as discussedabove. Advantageously, when applied to an oil well tripping system, thetop 131 of pipe 130 can be expected to pass through a region of interest306 which represents a portion of image data 304. Consequently, thefeature detection process used to detect the top 13 of pipe 130 may belimited to image data within region of interest 306.

FIG. 11B depicts a plot 310 (in dashed lines) showing the result of avertical projection wherein region of interest 306 is divided intovertical columns and the intensities of all of the pixels in each columnare added to arrive at a vertical projection value. Columns exhibiting alarge number of high intensity (white) pixels will have high verticalprojections values, whereas columns exhibiting a large number of lowintensity (black) pixels will have low vertical projection values. Inthe illustrated embodiment, each vertical column is one pixel wide.Accordingly, region of interest 306 is approximately 350 pixels wide(i.e. plot 310 spans 350 vertical projection columns). In otherembodiments, each column has a width comprising a plurality of pixels.Plot 310 may be low pass filtered to arrive at plot 312 (in solid line).In one particular embodiment, the low pass filter used to generate plot312 is a kaiser filter having a passband of 0-900 Hz and a cut-offfrequency of 2.5 kHz.

It can be seen from plots 310 and 312 that the vertical projectionexhibits three local minima which correspond to elevator components308A, 308B and to pipe 130. Controller 210 may interpret the centrallocal minimum A to represent an approximation of a vertical axis 314 ofpipe 130. Image processing component 212 may make use of a minimadetection algorithm to detect the central local minimum A. In someembodiments, elevator components 308A, 308B may be different. Thoseskilled in the art will appreciate that feature detection processes maydiffer where the expected features of the image (e.g. elevatorcomponents 308A, 308B) are different.

FIG. 11C depicts a plot 318 (in dashed lines) showing the result of ahorizontal projection wherein region of interest 306 is divided intohorizontal rows and the intensities of all of the pixels in each row areadded to arrive at a horizontal projection value. In the illustratedembodiment, each horizontal column is one pixel in height. Accordingly,region of interest 306 is approximately 550 pixels high (i.e. plot 318spans 550 horizontal projection rows). In other embodiments, each rowhas a height comprising a plurality of pixels. Plot 318 may be low passfiltered to arrive at plot 320 (in solid line). The low pass filter maybe the same as that used to generate the vertical projections.

In FIG. 11C, plot 320 exhibits a noticeable decay in region B, whichcorresponds to the vertical end 316 of pipe 130. In one particularembodiment, the region B decay is detected by convolving the plot 320horizontal projection with a curvelet representing an idealized decaysignal. Convolution is well known to those skilled in the art of digitalsignal processing. FIG. 11D exhibits such an idealized decay curvelet.The point along plot 320 where this convolution is a maximum may beselected as the vertical end 316 of pipe 130.

FIGS. 10-11D and the discussion presented above represent one embodimentof the signal processing of image processing component 212 for the imagedata corresponding to a single image sensor 202A, 202B, 202C. Thoseskilled in the art will appreciate that the same types of processing mayoccur for image data captured by other image sensors 202A, 202B, 202C tocapture three-dimensional information about the location of the top 131of pipe 130 and/or to add additional data to an estimate of the locationof the top 131 of pipe 130. The top 131 of pipe 130 may be used bycontroller 200 to determine the desired position d_(i) of end effectorN7 during tripping operations.

In accordance with another embodiment of the invention, image processingcomponent 212 performs a cross-correlation template matching operationbetween a selected subset of the image pixels and an idealized image (atemplate) containing the top 131 of pipe 130. The generalcross-correlation between two functions f and g is given by:

f ⊗ g = ∫_(−∞)^(∞)∫_(−∞)^(∞)f(u, v)g(u + x, v + y) 𝕕u 𝕕vand the normalized cross-correlation is given by:

$\frac{f \otimes g}{\sqrt{\int{\int{f^{2} \cdot {\int{\int g^{2}}}}}}} \leq 1$Generalizing this to two-dimensional discrete functions I_(ij) andB_(ij), the cross-correlation r is given by:

$r = \frac{\sum\limits_{i}{\sum\limits_{j}{( {I_{ij} - {\overset{\_}{I}}_{ij}} )( {B_{ij} - {\overset{\_}{B}}_{ij}} )}}}{\sqrt{\sum\limits_{i}{\sum\limits_{j}{( {I_{ij} - {\overset{\_}{I}}_{ij}} )^{2}{\sum\limits_{i}{\sum\limits_{j}( {B_{ij} - {\overset{\_}{B}}_{ij}} )^{2}}}}}}}$Here, r takes on a value between [−1,1] which can be used as a measureof a similarity between a selected portion of image data 204 (I_(ij))and data associated with an idealized template image (B_(ij)) containingthe top 131 of pipe 130.

FIG. 12 schematically depicts how this cross-correlation function r canbe used to detect a location of the top 131 of pipe 130 within imagedata 204. Image data 204 is parsed into a plurality of two-dimensionalimage portions 330. Image processing component 212 computes across-correlation r between the pixels (I_(ij)) of each portion 330 andthe pixels (B_(ij)) of a template image 332 containing the top 131 ofpipe 130. The portion 330 of image data 204 that exhibits the highestcross-correlation r with template image 332 (i.e. most closely matchestemplate image 332) is assumed to contain the top 131 of the pipe 130.

Advantageously, this cross-correlation template matching technique doesnot require that background scenery be removed from image data 204 (i.e.the preprocessing steps of FIG. 10 are not required). However, in somecircumstances, such as different light conditions (brightness andcontrast) for example, image preprocessing can be useful to improve theaccuracy and reliability of this cross-correlation template matchingtechnique. As with the feature detection technique of FIGS. 10A-10C and11A-11D, the computational resources consumed by this cross-correlationfeature matching technique may be reduced by performing the operationover a region of interest that occupies a subset of image data 204 (seeregion of interest 306 of FIG. 11A).

One variable which can impact this cross-correlation template matchingtechnique is the size of the horizontal and vertical jumps betweenneighboring image portions 330. For example, if the top left corner of afirst image portion 330 is at pixel (1,1), then a subsequent imageportion 330 may have a horizontal jump which may be as small as onepixel (i.e. a top left corner at pixel (2,1)) or the subsequent imageportion may have a larger horizontal jump. Similarly, the vertical jumpto a subsequent image portion 330 may be as small as one pixel (i.e. atop left corner at pixel (1,2)) or the vertical jump to the subsequentimage portion 330 may be larger. It will be appreciated that largerhorizontal and vertical jumps will result in a faster computation time,but may be more apt to lead to spurious results. In some embodiments,the horizontal and vertical jumps are in a range of [1, 10]. In otherembodiments, the horizontal and vertical jumps are in a range of [1, 4].In some embodiments, the cross-correlation template matching process isperformed in a number of iterations, wherein the horizontal and verticaljumps and the region of interest are decreased for each successiveiteration.

Other variables that influence this cross-correlation template matchingprocess include the possibility that pipe 130 moves off of the axis E11of elevator E6 (See FIG. 1). If the top 131 of pipe 130 moves away froma particular image sensor, then it will appear smaller in image data 204than in template image 332. Conversely, if the top 131 of pipe 130 movestoward a particular image sensor, then it will appear larger in imagedata 204 than in template image 332. This cross-correlation templatematching technique has been experimentally determined to reliably detectthe top 131 of pipe 130 for size differences of over 25%. A similarcomplication arises from the fact that pipe 130 may be suspended byelevator E6 at an angle that is different from the angle in which thepipe of template image 332 is suspended. This cross-correlation templatematching technique has been experimentally determined to reliably detectthe top 131 of pipe 130 for relative image rotation (i.e. between theactual image data 204 and template image 332) of over 5%.

The cross-correlation template matching technique presented aboverepresents one embodiment of the signal processing of image processingcomponent 212 for the image data corresponding to a single image sensor202A, 202B, 202C. Those skilled in the art will appreciate that the sametypes of processing may occur for image data captured by other imagesensors 202A, 202B, 202C to capture three-dimensional information aboutthe location of the top 131 of pipe 130 and/or to add additional data toan estimate of the location of the top 131 of pipe 130. The top 131 ofpipe 130 may be used by controller 200 to determine the desired positiond_(i) of end effector N7.

Image processing component 212 may also determine the angle at whichpipe 130 is oriented in order to determine the desired location d_(i) ofend effector N7. It will be appreciated by those skilled in the art thatif the location of the top 131 of pipe 130 is known (e.g. using one ormore of the techniques discussed above), then determining the locationof another point on the axis of pipe 130 will determine the angularorientation of pipe. For example, if the top 131 of pipe 130 is known intwo dimensions to have the coordinates (o_(x), o_(y)) and another pointon the axis of the pipe is known to have the coordinates (v_(x), v_(y)),then the angle of pipe 130 with respect to the horizontal axis is givenby α=tan⁻¹((o_(y)−v_(y))/(o_(x)−v_(x))).

FIGS. 13A-13C schematically depict one technique for obtaining a secondpoint on the axis of pipe 130. It is assumed that the top 131 point A)of pipe 130 has been determined (e.g. in accordance with one of theaforementioned techniques). Determining a second point B on the axis ofpipe 130 may be accomplished using a vertical projection, featurerecognition technique similar to that shown in FIG. 11B. The verticalprojections may be created by: creating a reduced size two-dimensionalmatrix 340 which is spaced below the top 131 (point A) of pipe 130 by afixed amount; dividing matrix 340 into vertical columns; and adding thevalues of all of the pixels in each column. Preferably, matrix 340 isrelatively small, particularly in the vertical dimension. In theillustrated embodiment, matrix 340 is 10 pixels high by 140 pixels wide.

FIG. 13B shows a vertical projection plot 342 similar to the verticalprojection plot 310 of FIG. 1B. FIG. 13C shows a plot 344 which is a lowpass filtered version of plot 342. FIG. 13C shows that plot 344comprises three local minima. The first and third minima correspond toelevator components 308A, 308B and the central minimum corresponds topoint B on pipe 130. Image processing component 212 may comprise a localminimum detection algorithm to locate the local minimum corresponding topoint B. In other embodiments, features other than local minima can beused to detect point B on pipe 130. For example, vertical projectionplot 324 may be convolved with an idealized curvelet to detect point B.Once the location of point B on pipe 130 is known, then image processingcomponent 212 may determine the angle of orientation of pipe 130 asdiscussed above.

It will be appreciated by those skilled in the art that signalpreprocessing steps similar to those of FIGS. 10A-10C may be used toincrease the accuracy of the vertical projection, feature detectiontechnique of FIGS. 13A-13C and to thereby increase the accuracy of thelocation of point B. Such preprocessing can be performed on the entireimage or on the reduced size matrix 340. In cases where the top 131(point A) of pipe 130 is determined by a cross-correlation templatematching technique (FIG. 12), a vertical projection, feature detectiontechnique (similar to FIGS. 13A-13C) may be performed on a reduced sizematrix to refine the location of the top 131 (point A) of pipe 130.

In accordance with another embodiment of the invention, an edgedetection technique combined with a Hough transform is used to locate asecond point (point B) on the axis of pipe 130. FIGS. 14A-14Cschematically depict how a subset 350 of image 204 is extracted for edgedetection. Subset 350 is preferably a relatively narrow matrix of pixelshaving an upper vertical boundary that corresponds (approximately) withthe top 131 (point A) of pipe 130. Subset 350 should be centeredhorizontally at point A and relatively narrow in width, so as not toinclude the other edges of elevator components 308A, 308B. Suchextraneous edges may make it difficult for the Hough transform toaccurately determine the angle of orientation of pipe 130. Subset 350 issubjected to an edge detection process to generate a binary image 352.The edge detection process may be a Roberts Cross, Sobel or Canny edgedetection process. These and other edge detection processes are known inthe art.

The use of a Hough transform to detect the angle of straight line(s)from binary edge detection data is known. In one particular embodiment,the Hough transform used for this process is the parametrictransformation ρ=x cos θ+y sin θ. This parametric transformation mapspoints (x_(i), y_(i)) in binary edge detection data 352 into sinusoidalcurves in the Hough domain (ρ, θ). Points (x_(i), y_(i)) that areco-linear in edge detection data 352 will intersect at a particularpoint (ρ, θ) in the Hough domain. This Hough angle θ may then be used todetect the angle α formed by pipe 130 with the horizontal axis accordingto α=90°−θ.

Edge detection data 352 exhibits two straight lines corresponding to theedges of pipe 130. This edge detection data 352 may generate two sets ofcurves in the Hough domain. Ideally, the members of the first set ofcurves should intersect one another in the Hough domain at points (ρ₁,θ₁) and the second set of curves should intersect one another in theHough domain at points (ρ₂, θ₂). However, since the edges of pipe 130are generally parallel, θ₁ should be substantially similar to θ₂. Insome embodiments, the Hough transformation process is carried on bothedges of pipe 130. In other embodiments, the Hough transformationprocess need only be carried out on a single edge. As is known in theart, the Hough domain may be divided into accumulator cells and peaks inthese accumulator cells may be interpreted as strong evidence that astraight line exists in edge detection data 352 which has Hough domainparameters within the accumulator cell.

Once the top 131 of pipe 130 and the orientation of pipe 130 are known,then image processing component 212 can use these parameters of pipe 130to determine the target position d_(i) of end effector N7 such that endeffector N7 can interact with pipe 130. This desired position d_(i) canthen be used by robot unit inverse kinematic component 214 and robotcontrol component 216 to generate appropriate control signals 206 forthe actuators of robotic system N2 as described above (see FIG. 8).

It may also be useful for controller 210 to use image data 204 todetermine abrupt changes in acceleration of pipe 130. Such abruptchanges can be indicative of pipe being lowered by elevator E6 into driptray E9 and the bottom of pipe 130 impacting drip tray E9. Once thebottom of pipe 130 impacts drip tray E9 (e.g. during a tripping outprocess), then robotic system N2 can be manipulated to make end effectorN7 grip pipe 130.

Abrupt changes in acceleration of pipe 130 may be detected using avertical projection feature detection technique (similar to that of FIG.11B), but on a different region of interest. Such a technique isschematically depicted in FIGS. 15A-15G.

FIGS. 15A-15B show image data 204 between time t1 and a later time t2,between which elevator E6 is lowering pipe 130. Region of interest 360is at the lower end of image 204, where the body of pipe 130 is distinctfrom the components of elevator E6. A vertical projection technique maybe used on region of interest 360 to determine the location of the bodyof pipe 130.

FIGS. 15C-15F show a low pass filtered vertical projection plot 362taken at time t1. The body of pipe 130 is determined to be located atlocal minimum D1. FIGS. 15E-15F also show a low pass filtered verticalprojection plot 364 taken at time t2. At time t2, the body of pipe 130is determined to be located at local minimum D2. Preprocessing similarto that of FIGS. 10A-10C may be used before implementing these verticalprojections. A minima detection algorithm or other feature detectionprocess may be used to locate points D1 and D2. Data from plots 362, 364may be used to calculate the acceleration of pipe 130 over time. FIG.15G shows a plot 366 of the acceleration of pipe 130 over time. Region368 of plot 366 shows a distinct change in acceleration of pipe 130.Accordingly, region 368 may be interpreted as being the time where pipe130 hits drip tray E9. The calculated acceleration may be subject to athresholding process to determine the time that pipe 130 impacts driptray E9.

FIG. 16A schematically depicts a method 400 of tripping out a pipe 130according to a particular embodiment of the invention. Method 400commences in block 410 and proceeds to block 412, where controller 210determines whether a pipe 130 is within the field of view of imagesensing system 202. This block 412 determination may be made byprocessing image data 204 from image sensing system 202, by interpretingdata from some other sensor (e.g. a sensor on elevator E6 whichdetermines when pipe coupler E8 has passed above racking platform N1) orby input of operator E10. If there is a pipe 130 within the field ofview of imaging system 202 (block 412 YES output), then method 400proceeds to block 414 where control system 200 waits for a sudden changein acceleration. The determination of a sudden change in accelerationmay be based on image data 204 and may be made using a thresholdingprocess, as described above. If a sudden change of acceleration isdetected (block 414 YES output), then system 200 may interpret this asoperator E10 manipulating the bottom of pipe 130 into drip tray E9.Method 400 then proceeds to block 416.

Blocks 416, 418 and 420 involve using image data 204 from image sensingsystem 202 to determine the location of the profile of pipe 130 (block416), to determine the orientation of pipe 130 (block 418) and, on thebasis of this information in combination with information from thesensors associated with robotic system N2, to controllably move roboticsystem N2 (block 420) such that end effector N7 moves toward a positionwhere in can grab pipe 130. This process may involve determining atarget position for end effector N7 and moving robotic system N2, so asto move end effector N7 toward this target position. The target positionfor end effector N7 is preferably dynamically updated using informationfrom image sensing system 202. When end effector is properly positionedto grab pipe 130 (block 422 YES output), then controller 210 causes endeffector N7 to grab pipe 130 in block 424. In block 426, controller 210causes robotic system N2 to controllably move end effector N7 to anappropriate location in rack N5 and to release pipe 130 in rack N5.Movement of robotic system N2 in block 426 may be done without feedbackfrom image sensing system 202.

FIG. 16B schematically depicts a method 500 for tripping in a pipe 130according to a particular embodiment of the invention. Method starts inblock 510 and then moves to block 512, where controller 210 causesrobotic system N2 to move such that end effector N7 is in position tograb a pipe 130 from rack N5. Controller 210 then causes end effector N7to grab a pipe in block 514 and begins to move robotic system N2 towardthe field of view of image sensing system 202 in block 516. Movement ofrobotic system N2 in blocks 510 and 514 may occur without feedback fromimage sensing system 202. Once pipe 130 is located in the field of viewof image sensing system 202, then image data 204 is obtained andcontroller 210 uses this image data in combination with information fromthe sensors associated with robotic system N2 to move the top of pipe130 into alignment with the axis E11 of elevator E6.

In the illustrated embodiment, controller 210 determines the location ofthe profile of pipe 130 using image data 204 (in block 518) and causesrobotic system N2 to move end effector N7 in response to thisinformation in combination with information from the sensors associatedwith robotic system N2 (in block 520). In the block 522 movement ofrobotic system N2, the target position of end effector N7 may be thetarget position required to place the top of pipe 130 in alignment withelevator axis E11. This target position may be dynamically updated onthe basis of image data 204. When it is determined (based on image data204) that the top of pipe 130 is located in alignment with axis E11 ofelevator E6 (block 522 YES output), then elevator E6 grabs pipe 130 inblock 524. Once elevator E6 has grabbed pipe 130, then controller 210may cause end effector N7 to release pipe 130 in block 526. Pipe 130 canthen be lowered into the oil well by elevator E6.

As briefly discussed above, in some embodiments system 10 may be usedwithout any machine vision system. An example of the operation of suchan embodiment is discussed in the following paragraphs with reference toFIGS. 17, 18 and 19A-C.

FIG. 17 schematically depicts a system controller 600 for a roboticsystem 602 such as, for example, system 10 of FIGS. 1-5C describedabove. Robotic system 602 comprises a plurality of actuators 602A foreffecting movement of the components of system 602, and a plurality ofsensors 602B for providing positional information about the componentsof system 602. Controller 600 is similar to controller 210 describedabove with reference to FIGS. 8 and 9, except that instead of anymachine vision system, controller 600 comprises a memory storingpositional information 604 coupled to a processor 606. Processor 606 maydetermine the target position d_(i) of end effector based on positionalinformation 604 and input from an operator who may indicate that a pipe130 is ready to be grabbed from an elevator axis (for a tripping outoperation) or pipe rack (for a tripping in operation), as describedbelow. Controller 600 comprises a robot unit inverse kinematic component608, which processes target position d_(i) to obtain a set of desiredcoordinates q_(d) for robotic system 602 (in the measurement space ofthe position sensors of robotic system 602). Comparison component 610then compares the desired coordinates q_(d) for robotic system 602 tothe actual robot unit coordinates q (i.e. robot unit position datasensed by the sensors of robotic system 602). Robot control component612 then uses the differences between the actual coordinates q and thedesired coordinates q_(d) to generate appropriate control signals 614for the actuators of robotic system 602.

FIG. 18 schematically depicts a method 700 for tripping out a pipe 130according to a particular embodiment of the invention. Method 700 may becarried out, for example, by a system such as system 10 of FIGS. 1-5Cdescribed above, under control of a suitably programmed systemcontroller, such as, for example, controller 600 of FIG. 17. Method 700commences in block 710 and proceeds to block 712, where a pipe 130 israised by elevator E6 and unscrewed from the pipe(s) remaining in thewell, as described above. Method 700 then proceeds to block 714, wherecontroller 600 causes end effector N7 to grab pipe 130 while pipe 130 isstill oriented along elevator axis E11, as shown in FIG. 19A. Positionalinformation 604 may comprise information specifying the position ofelevator axis E11 to facilitate the grabbing of pipe 130 by end effectorN7.

Next, method 700 proceeds to block 716, where, a human drill headoperator E10 (FIG. 1) guides the lower end of pipe 130 over drip trayE9, as shown in FIG. 19B. Controller 600 may facilitate such movement ofthe lower end of pipe 130, for example, by allowing end effector N7 tobe moved by the movement of the lower end of pipe 130 (referred toherein as “zero torque mode”), or by responding to torque detected bysensors of robotic system N2 to assist the movement of pipe 130(referred to herein as “torque feedback mode”) by moving end effector N7to reduce the torque exerted on robotic system N2 due to the movement ofthe bottom portion of pipe 130. When the lower end of pipe 130 ispositioned over the drip tray E9, the orientation of pipe 130 is nolonger vertical, and elevator E6 may be displaced some distance awayfrom elevator axis E11 in an opposite direction from drip tray E9.

Next, method 700 proceeds to block 718, where elevator E6 is lowered byoperator E10 such that pipe 130 rests on drip tray E9, and elevator E6is detached from pipe 130. Detaching of elevator E6 could be effected byoperator E10 or triggered by one or more sensors in drip tray E9. Justprior to detaching elevator E6, controller 600 may cause end effector N7to pull back a short distance from elevator axis E11 toward drip trayE9, such that elevator E6 is more closely aligned with elevator axis E11and swinging of elevator E6 is reduced or eliminated.

Next, method 700 proceeds to block 720, where controller 600 causes endeffector N7 to return to a “home” position with pipe 130, as shown inFIG. 19C. The home position may be achieved, for example, by retractingarm N6 such that end effector N7 is as close as possible to mast 104with arm N6 and end effector N7 aligned along a line between mast axis117 and elevator axis E11. Positional information 604 of controller 600may store information specifying the home position.

Next, method 700 proceeds to block 722, where controller 600 causes endeffector N7 to manipulate pipe 130 to the open end of rack N5, as shownin FIG. 19D, and then push pipe 130 into its racking location.Controller 600 may, for example, cause end effector N7 to move pipealong a predetermined path from the home position to the rackinglocation of pipe 130, as specified by information stored in positionalinformation 604. The racking location for pipe 130 preferablycorresponds to a location of the bottom of pipe 130 in drip tray E9.Next, method 700 proceeds to block 724, where controller causes endeffector N7 to release pipe 130 when pipe is in its racking location,and then return to the home position to prepare for the next trippingoperation. Method 600 then ends at block 726.

FIGS. 20A and 20B schematically depict an elevator E6 according to oneembodiment of the invention. Elevator E6 comprises a pipe coupler E8comprising two collar portions E8A and E8B pivotally coupled together bya pipe coupler pivot joint E8C. A locking mechanism E8D is operable toselectively lock collar portions E8A and E8B in a closed position shownin FIGS. 20A and 20B. The details of construction of collar portions E8Aand E8B, pipe coupler pivot joint E8C and locking mechanism E8D areknown in the art, and are not specifically illustrated or described indetail.

In the embodiment of FIGS. 20A and 20B, extension flanges E6A, E6B andE6C are respectively coupled to collar portions E8A and E8B and pipecoupler pivot joint E8C. A pipe coupler actuator E6D is connectedbetween extension flanges E6B and E6C, such that movement of pipecoupler actuator E6D into an extended position forces collar portionsE8A and E8B together into the closed position shown in FIGS. 20A and20B, and movement of pipe coupler actuator E6D into a retracted positionforces collar portions E8A and E8B apart (if locking mechanism E8D isnot locked) into an open position (not shown). Pipe coupler actuator E6Dmay comprise, for example, a pneumatic cylinder, and may include one ormore sensors E6H for providing a system controller of a robotic systemsuch as those discussed above with an indication of when pipe coupleractuator E6D is in the extended position or the retracted position. Theoperation of pipe coupler actuator E6D may be controlled by the systemcontroller. Valves may also be provided to allow manual operation ofpipe coupler actuator E6D.

A locking mechanism actuator E6E is connected between extension flangeE6A and locking mechanism E8D, such that movement of locking mechanismactuator E6E into an extended position forces locking mechanism E8D intoa locked position as shown in FIGS. 20A and 20B, and movement of lockingmechanism actuator E6E into a retracted position forces lockingmechanism E8D into an unlocked position (not shown). When lockingmechanism E8D is in the unlocked position, collar portions E8A and E8Bmay be moved apart into an open position (not shown). Locking mechanismactuator E6E may comprise, for example, a pneumatic cylinder, and mayinclude one or more sensors (not specifically enumerated) for providingthe system controller with an indication of when locking mechanismactuator E6E is in the extended position or the retracted position. Theoperation of locking mechanism actuator E6E may be controlled by thesystem controller. Valves may also be provided to allow manual operationof locking mechanism actuator E6E.

Elevator E6 may also comprise a tilting actuator (not shown) tofacilitate tilting of elevator E6 to allow pipe coupler E8 to beattached to a horizontally oriented pipe. The tilting actuator maycomprise, for example, a pneumatic cylinder. The tilting actuator may becontrolled by the system controller, or manually.

A pipe presence sensor E6F (FIG. 20B) may be attached to one of collarportions E8A and E8B for providing the system controller with anindication of when a pipe is located between collar portions E8A andE8B. In the illustrated embodiment, pipe presence sensor E6F comprises amechanical switch E6G which is activated when a pipe is located betweencollar portions E8A and E8B. Alternatively or additionally, pipepresence sensor E6F could comprise one or more of a laser sensor, anultrasonic sensor or a magnetic sensor.

In operation, elevator E6 may be controlled by the system controller inconjunction with the operation of a robotic system for manipulatingpipes such as, for example, robotic system N2 (or 602) described above.The system controller may provide control signals and receive feedbacksignals from the actuators and sensors of elevator E6 though a wirelessconnection such as, for example, a radio frequency (RF) connection. Intripping out operations, elevator E6 may be controlled to maintaincollar portions E8A and E8B in the closed position with lockingmechanism E8D in the locked position until the system controllerreceives confirmation from the sensors of robotic system N2 that a pipeheld by elevator has been successfully grabbed by end effector N7.Conversely, in tripping in operations, robotic system N2 may becontrolled to maintain grabbing members N7A and N7B of end effector inthe closed position until the system controller receives confirmationfrom the sensors of elevator E6 that a pipe held by end effector N7 hasbeen successfully received in pipe coupler E8 and collar portions E8Aand E8B are in the closed position with locking mechanism E8D in thelocked position.

While a number of exemplary aspects and embodiments have been discussedabove, those of skill in the art will recognize certain modifications,permutations, additions and sub-combinations thereof. For example:

-   -   There are other applications where it is desirable to reduce or        eliminate human involvement in re-orienting, guiding,        positioning and racking of elongated objects. Solutions which        reduce or eliminate human involvement in tripping out and        tripping in operations for oil well servicing may also be        suitable use in these other applications.    -   Racking platform N1 may optionally comprise a safety railing N3        which may be portable and removable from racking platform N1.    -   In some of the embodiments described above, image processing        component 212 makes use of image data 204 to determine the        location of the end 131 of pipe 130 during tripping operations.        In other embodiments, other sensors, such as ultrasound sensors,        radar sensors, sonar sensors and laser proximity sensors, may be        used in addition to or in the alternative to image sensors.    -   In one particular embodiment described above, image processing        component 212 performs a template matching technique to detect        the top 131 of pipe 130. In other embodiments, template matching        techniques may be employed which use other vector distance        formula (i.e. other than cross-correlation) to provide an        estimate of the data that best matches a given template.    -   The description set out above provides a number of example        methods which may be used to process image data 204 to detect        the top 131 of pipe 130. Those skilled in the art will        appreciate that there are other techniques which could be used        to process image data 204 to detect the top 131 of the pipe 130.        For example, a Hough transformation method could be used to        detect the top 131 of pipe 130. The invention should be        understood to include such techniques in addition to (or as        alternatives to) the techniques described herein.    -   The description set out above provides a number of example        methods which may be used to process image data 204 to detect a        second point on pipe 130 and/or the orientation of pipe 130.        Those skilled in the art will appreciate that there are other        techniques which could be used to process image data 204 to        detect the second point on pipe 130 and/or the orientation of        pipe 130. For example, a template matching method could be used        to detect the second point on pipe 130 and/or the orientation of        pipe 130. The invention should be understood to include such        techniques in addition to (or as alternatives to) the techniques        described herein.    -   The description set out above provide an example technique which        may be used to process image data 204 to detect rapid changes in        acceleration of pipe 130. Those skilled in the art will        appreciate that there are other techniques which could be used        to process image data 204 to detect rapid acceleration changes        in pipe 130. The invention should be understood to include such        techniques in addition to (or as alternatives to) the techniques        described herein.    -   The description set out above refers to tripping pipes in and        out of an oil well, but the invention may also have application        to tripping portions of a drill string or other elongated        objects in and out of wells.

It is therefore intended that the following appended claims and claimshereafter introduced are interpreted to include all such modifications,permutations, additions and sub-combinations as are within their truespirit and scope.

1. A robotic system coupled to a racking platform of an oil well serviceor drilling rig, the robotic system comprising: a base coupled to theracking platform at a fixed location; a mast pivotally coupled to thebase by a mast pivot joint allowing rotation of the mast about a mastaxis; a mast actuator for controllably rotating the mast about the mastpivot joint; an arm coupled to the mast, the arm including proximal anddistal ends, wherein the distal end is moveable along a radial directionwith respect to the mast axis, and wherein the proximal end is moveablealong an axial direction with respect to the mast axis; an arm actuatorfor controllably moving the arm along the radial direction; an endeffector pivotally coupled to the distal end of the arm by an endeffector pivot joint allowing rotation of the end effector about an endeffector axis oriented at least substantially parallel to the mast axis,the end effector comprising at least one grabbing member operable toselectively grab an elongated object under control of a grabbing memberactuator; and an end effector actuator for controllably rotating the endeffector about the end effector pivot joint.
 2. The robotic system ofclaim 1, wherein the base is coupled to the racking platform by a basepivot joint for allowing rotation of the base about an axis at leastsubstantially perpendicular to the mast axis, the robotic systemcomprising a base actuator for controllably moving the base between anoperational position wherein the mast axis is oriented at leastsubstantially perpendicularly to a plane of the racking platform, and astorage position wherein the mast axis lies at least substantiallywithin the plane of the racking platform.
 3. The robotic system of claim1, wherein the arm comprises a plurality of segments pivotally coupledto one another, and wherein a first end of a first segment is connectedto the arm actuator, and a first end of a second segment is connected tothe mast, such that movement of the first end of the first segmenttoward the first end of the second segment causes the arm to extendoutwardly from the mast along the radial direction.
 4. The roboticsystem of claim 1, wherein the end effector comprises two opposedgrabbing members each coupled to a housing of the end effector by atleast one fixed pivot joint, the grabbing members moveable between aclosed position and an open position under control of the grabbingmember actuator.
 5. The robotic system of claim 4, wherein the fixedpivot joints comprise a plurality of shock absorbing bushings.
 6. Therobotic system of claim 4, wherein the grabbing member actuatorcomprises an extendable member, and the opposed grabbing members arecoupled to the extendable member by a pair of pivoting links that arepositioned opposed to any opening of the grabbing members when thegrabbing members are in the closed position.
 7. The robotic system ofclaim 4, wherein each grabbing member comprises a detachable grabbingportion configured to grab a pipe having a predetermined diameter, suchthat the end effector may be adapted to grab a plurality of pipes havingdifferent diameters by providing different detachable grabbing portions.8. The robotic system of claim 1, comprising a controller forcontrolling the operation of the mast actuator, the arm actuator, theend effector actuator and the grabbing member actuator, the controllercomprising a processor coupled to a memory storing positionalinformation for manipulating pipes into and out of the racking platform.9. The robotic system of claim 8, comprising a plurality of sensors forproviding the controller with information about the orientations of themast, arm, end effector and at least one gripping member.
 10. Therobotic system of claim 1, wherein the mast actuator, the arm actuatorand the end effector actuator comprise servo motors.
 11. The roboticsystem of claim 10, wherein the grabbing member actuator comprises astepper motor.
 12. The robotic system of claim 1, further comprising apositional information storing system.
 13. A mobile apparatus for oilwell servicing or drilling, the apparatus comprising: a mobile platform;a derrick pivotally coupled to the mobile platform and moveable betweena deployed position and a storage position; a racking platform coupledto the derrick, the racking platform defining a plurality of elongatedobject receiving locations; an elevator supported from the derrick forraising and lowering elongated members along an elevator axis; and, arobotic system coupled to the racking platform at a fixed location, therobotic system comprising a mechanism having at least three degrees offreedom for manipulating an upper portion of an elongated member withina plane at least substantially parallel to a plane of the rackingplatform, wherein the robotic system comprises: a mast coupled to theracking platform at the fixed location by a mast pivot joint allowingrotation of the mast about a mast axis oriented at least substantiallyperpendicularly to the racking platform; an arm coupled to the mast, thearm including proximal and distal ends, wherein the distal end ismoveable along a radial direction with respect to the mast axis, andwherein the proximal end is moveable along an axial direction withrespect to the mast axis; an arm actuator for controllably moving thearm along the radial direction; an end effector pivotally coupled to thedistal end of the arm by an end effector pivot joint allowing rotationof the end effector about an end effector axis oriented at leastsubstantially parallel to the mast axis, the end effector comprising atleast one grabbing member operable to selectively grab an elongatedobject under control of a grabbing member actuator; and an end effectoractuator for controllably rotating the end effector about the endeffector pivot joint.
 14. The apparatus of claim 13, wherein the rackingplatform is pivotally coupled to the derrick, and wherein the roboticsystem is pivotally coupled to the racking platform at the fixedlocation, such that the racking platform and the robotic system aremoveable into at least substantially parallel orientations with respectto the derrick when the derrick is in the storage position.
 15. Theapparatus of claim 13, wherein the racking platform comprises: a frame;a plurality of finger members mounted on the frame, wherein a pair ofadjacent finger members defines an elongated object receiving paththerebetween, and wherein a first one of the pair of adjacent fingermembers comprises a plurality of arcuate indentations defining theelongated object receiving locations along an edge thereof; and aplurality of toggle locks mounted on pivot joints on a second one of thepair of adjacent finger members, the toggle locks coupled incomplementary pairs biased into a predetermined angular relationshipwith one another such that when one of the toggle locks of acomplementary pair is pivoted out of the elongated object receiving paththe other of the toggle locks in the complementary pair is urged intothe elongated object receiving path, wherein a last complementary pairof toggle locks comprises a biasing mechanism configured to bias a lasttoggle lock closest to the frame into the elongated object receivingpath.
 16. The apparatus of claim 13, wherein the elevator comprises: anelongated object coupler for selectively engaging an upper portion of anelongated object, the elongated object coupler moveable between an openposition and a closed position; an elongated object coupler actuator formoving the elongated object coupler between the open position and theclosed position; and an elongated object coupler sensor for producing anindication of whether the elongated object coupler is in the openposition or the closed position.
 17. The apparatus of claim 16, whereinthe elevator comprises: a locking mechanism for selectively locking theelongated object coupler in the closed position, the locking mechanismmoveable between a locked position and an unlocked position; a lockingmechanism actuator for moving the locking mechanism between the lockedposition and the unlocked position; and a locking mechanism sensor forproducing an indication of whether the locking mechanism is in the openposition or the closed position.
 18. The apparatus of claim 17, whereinthe elevator comprises an elongated object presence sensor for producingan indication of whether the upper portion of an elongated object isengaged by the elongated object coupler.
 19. A method of removing anelongated object from an oil well, the method comprising: providing anapparatus according to claim 13; raising the elongated object along theelevator axis with the elevator; grabbing an upper portion of theelongated object with the robotic system while the elongated object islocated along the elevator axis; allowing a bottom portion of theelongated object to be moved over a tray located below the rackingplatform; lowering the elevator such that a bottom end of the elongatedobject rests on the tray at a location corresponding to a selected oneof the elongated object receiving locations defined by the rackingplatform; and moving the upper portion of the elongated object to theselected one of the elongated object receiving locations defined by theracking platform.
 20. The method of claim 19, wherein allowing thebottom portion of the elongated object to be moved comprises allowingthe robotic system to be moved by torque exerted thereon due to movementof the bottom portion of the elongated object, or detecting torqueexerted on the robotic system due to movement of the bottom portion ofthe elongated object and assisting the movement of the bottom portion ofthe elongated object by moving the robotic system to reduce the torqueexerted thereon, or both.
 21. The method of claim 19, wherein moving theupper portion of the elongated object to the selected one of theelongated object receiving locations comprises returning the roboticsystem to a home position and then moving the robotic system along apredetermined path from the home position to the selected one of theelongated object receiving locations.